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(54) Title: METHOD AND APPARATUS FOR ELECTRONICALLY DISTRIBUTING MOTION PANORAMIC IMAGES 

(57) Abstract 

The present invention 
discloses an electnxiic 
image distribution apparatus 
for distributing motion 
panoramic images. The main 
portion of the apparatus 
is a motion panoramic 
image server that has a 
source of motion panoramic 
images. The source of the 
panoramic images frames 
may be a motion panoramic 
image playback device 
or a motion panoramic 
camera system (405). The 
motion panoramic image 
server (400) transforms the 
panoramic image frames 
into an intermediate format 
that is more conducive 
for electronic transmission 
(417). The motion 

panoramic image server 
(400) then transmits the 
motion panoramic images to 
client systems (495. 497). 
A first transmission system 
divides the intermediate 

planar representation into ^ ^ ..^a aa^^ 

individual dies (441. 445) and transmits tiles that are requested by clients systems (440. 443). 
system divides the intermediate planar representation into individual tiles on different multicast 
subscribe to the multicast channels needed to render an image in the direction of a desired view. 
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A second panoramic image transmission 
channels. Client systems (495, 497) then 
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Method and Apparatus For Electronically 
Distributing Motion Panoramic Images 



FTFI D OF THF INVENTION 

The present invention relates to the field of image processing and data 
distribution. In particular the present invention discloses methods and apparatus for 
processing, transmitting, and post-processing panoramic image information. 

R ArKOROUNH OF THE IN VENTION 

Most cameras only record a view vi^ithin a small viewing angle. Thus, a 
typical conventional camera only captures an image in the direction that the camera is 
aimed. Such conventional cameras force viewers to look only at what the camera operator 
choo.ses to focus on. 

Panoramic cameras capture a much wider field of field. A panoramic camera 
con.structed using an optical "fish-eye" lens may capture at most a 210 degree field of view. 
A panoramic camera con.structed with a caladioptric lens captures a full 360 degree 
panoramic image. With a large panoramic field of view, individual panoramic image 
viewers can decide what they will focus on. 

Conventional images are often delivered by electronic means. For example, 
television and the Internet deliver conventional images across wired and wireless electronic 
media. However, there are no standard means of delivering panoramic images 
electronically. Since panoramic images are so large, it is difficult to deliver panoramic 
images using conventional image transmission techniques. To further compound the 
problem, motion panoramic images require a very high bandwidth channel for electronic 
distribution. Thus, it would be desirable to have an electronic distribution system that 
efficiently distributes motion panoramic image information. 
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SUMMARY OF THE INVENTION 

The present invention discloses an electronic image distribution apparatus for 
distributing motion panoramic images. The main portion of the apparatus is a motion 
panoramic image server that has a source of motion panoramic images. The source of 
panoramic image frames may be a motion panoramic image playback device or a connection 
to a motion panoramic camera system. The motion panoramic image server transforms the 
panoramic image frames into an intermediate planar representation that is more conducive 
for electronic transmission. The motion panoramic image server then transmits the motion 
panoramic images to client systems. The present invention proposes two different 
transmission systems. 

A first panoramic image transmission system divides the intermediate planar 
representation into individual tiles. Individual client systems then request the individual 
tiles needed to construct a view into the panoramic image on the client system. The server 
then u-ansmits the requested tiles to the client system. A highly compressed full panoramic 
image is transmitted occasionally to handle a situation where a user has moved the view into 
the panoramic image to a position where the client system does not have the necessary tiles 
for creating a view. 

A second panoramic image transmission system also divides the intermediate 
planar representation into individual tiles. However, the second panoramic image 
u-ansmission system then transmits the individual tiles on different multicast channels. 
Client systems then subscribe the multicast channels needed to render an image in the 
direction of a desired view. 

Other objects, features and advantages of present invention will be apparent 
from the company drawings and from the following detailed description. 
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RPfFF DFSCRIPTIQN OF t hF DRAWINGS 

The objects, features and advantages of the present invention will be apparent 
to one skilled in the art, in view of the following detailed description in which: 

Figure 1 illustrates one embodiment of a panoramic camera system that 
creates annular images. 

Figure 2a illustrates an annular image that is recorded by the panoramic 
camera sysiem of Figure 1 . 

Figure 2b illustrates how the annular image of Figure 2a appears after it has 
been unwrapped by geometrical transformation software. 

Figure 3 illustrates the 360 degree band that the panoramic camera system of 
Figure 1 captures and Figure 3 also illastraies a cylindrical projection data format. 

Figure 4 illustrates a networked computer systems arrangement used to 
distribute motion panoramic .mages captured by the panoramic camera system Figure 1 
from a panoramic image server to client systems. 

Figure 5A illustrates a conceptual view of a spherical view captured by the 
panoramic camera sysiem of Figure 1. 

Figure 5B illu.stralcs a panoramic image frame in a spherical intermediate 
planar representation that has been unwrapped and divided into individual tiles. 

Figure 5C illustrates a panoramic .mage frame in a cylindrical intermediate 
planar represemation that has been divided into individual tiles or a spherical intermediate 
planar repre.sentation that has been adjusted and divided mto t.Ies. 

--3-- 
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Figure 5D illustrates a pianoramic image frame in a low resolution 
termediate planar representation that has been divided into individual tiles. 



Figure 5E illustrates a panoramic image frame in a medium resolution 
intermediate planar representation that has been divided into individual tiles. 

Figure 5F illustrates a panoramic image frame in a high resolution 
intermediate planar representation that has been divided into individual tiles. 

Figure 5G illustrates a panoramic annular image frame that has been divided 
into spatially variant tiles. 

Figure 6A illustrates a view within a panoramic image frame in an 
intermediate planar representation. 

Figure 6B illustrates four tiles from Figure 6A transmitted to construct a 
view within a client system. 

Figure 7A illustrates a view within a panoramic image frame intermediate 
planar representaiion that has suddenly shifted by a large angular value. 

Figure 7B illustrates a view within a panoramic image frame intermediate 
planar representation that has suddenly shifted by a small angular value. 

Figure 8 illustrates a flow diagram that describes how a panoramic image 
client system negotiates a connection with a motion panoramic image server. 

Figure 9 illustrates a flow diagram that describes how a motion panoramic 
image server sends image tiles to a panoramic image client 
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Figure 10 illustrates a flow diagram that describes how a motion panoramic 
in,age client renders a panoramic image v.ew using image tiles received from a server. 

Figure 11 illustrates a panoramic image frame in an intermediate planar 
representation that has been divided into overlapping tiles. 

Figure 12 illustrates a networked computer arrangement wherein a 
panoramic image se.er distributes motion panoramic image tiles on individual mult.cast 

channels. 

Figure 13 illusimes a now diagram lha. describes how a motion panoramic 
i„age Cien, snb.scribe. >o muUicas, panoramic rmage .ile channels and renders an rmage 
using received image tiles. 

Figure 14 illustrates a panoramic annular image divided into quadrants that 
are served on different multicast channels. 

Figure 15 illustrates a panoramic image server that transmits panoramic 
i.age information on different multicast channels on a mult.cast backbone and cUent 
systems that subscribe to the multicast channels. 



wo 00/08889 



PCTAJS99/01I04 



nFTAIl ED DFSCRIPTION OFTHF PREFERRED EMBODIMENT 

A method and apparatus for compressing and distributing motion panoramic 
images is disclosed. In the following description, for purposes of explanation, specific 
nomenclature is set forth to provide a thorough understanding of the present invention. 
However, it will be apparent to one skilled in the art that these .specific details are not 
required in order to practice the present invention. For example, the present invention's 
method of distributing panoramic images has been described with reference to the Internet. 
However, other types of digital information distribution systems such as private data 
networks and satellite systems may be u.sed with the teachings of the present invention. 



The Basic Panoramic Camera Design 



Figure 1 illustrates an embodiment of one possible 360 degree panoramic 
camera system. The image capture mechanism 150 of the panoramic camera system 
captures an angular image of the surrounding panorama that is reflected off the main 
reflector 110. More details about the panoramic camera system of Figure 1 can be found in 
the copending patent application entitled "Panoramic Camera"., filed June 1 1, 1997 and 
having serial number 08/872,525 which is hereby incorporated by reference. 

Figure 2a illustrates an example of a panoramic image captured as an annular 
image. As illustrated in Figure 2a. the surrounding panorama is distorted by the main 
reflector having radial symmetry such that the surrounding represented as a two-dimensional 
annular image. In one embodiment, the shape of the main reflector transforms the .standard 
rectangular coordinate .system of the image into a polar coordinate system. To view the 
panoramic image captured as a two-dimensional annular image, the annular image may be 
unwrapped to create an intermediate planar representation. The unwrapping is performed 
using a geometric transformation operation. Figure 2b illustrates how the annular image of 
Figure 2b appears after the images are geometrically transformed from the annular 
representation to a rectangular intermediate planar representation. 
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The panoramic camera system of Figure 1 captures an entire 360 degree 
image of the surrounding horizon as illustrated in Figure 3. The panoramic camera system 
captures a panoramic image that includes objects as low as 50 degrees below the horizon 
and objects as high as 50 degrees above the horizon. However, alternate panoramic camera 
systems may capture a full spherical panoramic view, a partial 180 degree hemispherical 
view, or any other panoramic view. Furthermore, a panoramic image may be constructed by 
combining a mosaic of convention images captured with convention camera systems. For 
example, eight convention cameras can be used to capture images from eight different 
directions. Those images are then combined togetiier to create a single panoramic image. 

A Motion Panoramic Image Capture System 

When the image capture mechanism 150 of the panoramic camera of Figure 
1 is a motion image capture mechanism such as a charged coupled device (CCD) or a film- 
based motion picture image capture system, Uien it is possible to capture a series of 
panoramic annular images. The series of panoramic annular images can be used to generate 
a motion panoramic display. Figure 4 illustrates one possible embodiment of an electronic 
motion panoramic image capture system. 

Referring to Figure 4, a panoramic camera system 405 captures a series of 
panoramic annular image frames with an electronic image capture mechanism. For 
example, the electronic image capture mechanism can be a charged coupled device image 
capture mechanism or CMOS image sensor that generates digitized panoramic amiular 
image frames. The panoramic annular image frames are transmitted from panoramic camera 
system 405 to a panoramic image server 400 across a high bandwidth communication link 
402. The communication link must be high bandwidtfi since each panoramic amiular image 
frame contains a very large amount of information and to provide a flicker-free motion 
image, several panoramic images must be transmitted each second. 

Alternate embodiments of motion panoramic image capture systems can also 
be used. For example, a motion panoramic capture system may be created with two motion 
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picture camera systems pointing in opposite directions wherein each motion picture camera 
system is equipped with a fisheye lens that captures at least a 180 degree hemispherical field 
of view. In such a panoramic image system, two distorted fisheye images from each 
hemispherical view would need to be '^stitched" together in to created a single panoramic 
image. 

A First Motion Panoramic Image Distribution System Embodiment 

As previously set forth, the full panoramic annular image frames generated 
by the panoramic camera system (such as panoramic camera system 405 of Figure 4) are so 
large thai a very high bandwidth link is needed to transmit full motion images to a recipient. 
Such high bandwidth links are rare and expensive. Thus, for mass distribution of panoramic 
imagery, more bandwidth efficient methods of distributing panoramic images are needed. 

The present invention introduces two different methods of distributing 
motion panoramic images, A first method divides panoramic images into tiles and transmits 
only the tiles needed to construct a desired image. A second method, described in a later 
section, divides panoramic images into individual tiles and then transmits all the tiles on 
different multicast channels. 

The Selective Tile Transmission Svstem 

To efficiently transmit a panoramic image, the present invention first 
proposes a selective tile transmission system. In the selective tile transmission system, a 
server computer system divides each panoramic frame into tiles. The tile size and shape are 
preferably selected in such a manner that is dependent upon a coordinate system used by the 
format used to transmit the tiles. Then, the tiles needed to construct an image on a client 
computer system are transmitted from the server system to individual clients. 

In a preferred embodiment, the server system transfonns each incoming 
panoramic image frame into an intermediate planar representation. This transformation step 
is not necessary but provides certain advantages. The selective tile transmission system will 
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be described with reference to a system that performs this transformation in Figures 4, 5, 6, 
7. 8 and 9. 

Referring to the particular embodiment of the selective transmission system 
illustrated in Figure 4, a panoramic annular image transformation system 417 "unwraps" 
annular panoramic image frames into an intermediate planar representation. The 
intermediate planar representation is more convenient for generaUng images suitable for 
viewing. Specifically, the intermediate format should require fewer calculations to generate 
a final rectangular projection. 

In one embodiment, the incoming annular panoramic image data is 
geometrically transformed into a spherical coordinate system with a spherical projection 
representation. In such an embodiment, a single frame of the image data may be viewed 
with the Live Picture viewer from Live Picture. Inc. Figure SA conceptually illustrates a 
spherical data format wherein the view of the surroundings is projected onto a sphere 
surrounding a view point. 

In another embodiment, the incoming annular panoramic image dau is 
geometrically transformed into a cylindrical coordinate system in a cylindrical projection 
representation. An image transformed into a cylindrical coordinate system may be viewed 
witii the QuickTime VR viewer created by Apple Computer, Inc. Figure 3 conceptually 
illustrates a cylindrical data format wherein the view of the surroundings is projected onto a 
cylinder surrounding a view point. 

After converting the panoramic image frames into an intermediate planar 
representation, the system of tiie present invention divides the intermediate planar 
representation into individual tiles. In Uie present invention, the size of the tiles are 
dependent upon the coordinate system of the intermediate planar representation. In such 
embodiments, the tiles may be spatially variant. 
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The size of the tiles may further be dependent upon the client system that will 
be displaying the panoramic image data. In such an embodiment, each client uses a 
different process (441 or 445) to divide intermediate planar representations into tiles since 
each client may use different client sizes. However, in a standardized embodiment, the size 
of the tiles are fixed such that a single process (illustrated as dotted box 449) may be used to 
divide an intermediate planar representation into individual tiles. 

Figure 5A illustrates a conceptual diagram of the spherically encoded 
intermediate planar representation. As illustrated in Figure 5A, the spherically encoded 
intermediate planar representation represents the surrounding world as image data mapped 
onto a sphere. To divide and organize the data, the sphere is broken into individual bands 
on sphere A, B, C, D, E, and F. Note that the bands near the "equator'' of the spherically 
encoded intermediate planar representation contain more data than the bands closer to the 
"poles." 

In one embodiment of the present invention, the image data is represented 
individual bands that are divided into individual tiles as illustrated in Figure 5B. Note that 
there will be fewer tiles in the bands from higher inclination angles. Alternatively, there 
will be the same number of tiles, but the tiles will contain data from a smaller yaw angles. If 
the data from the smaller yaw angles is increased, this type of intermediate data format can 
be graphically illustrated as show in Figure 5C. Note that the graphical data representation 
of Figure 5C also matches the cylindrical data format of Figure 3. Although the data 
format of Figure 5B may be used and provides certain advantages such as a more compact 
data format, this document will focus on the format of Figure 5C since it simplifies the 
description. 

Referring to the particular embodiment of Figures 5A, 5B, and 5C, a first 
row of tiles covers from the horizon line to thirty degrees above the horizon. A second row 
of tiles covers from thirty degrees above the horizon to forty-two degrees above the horizon. 
A third, top, row of tiles covers the band from forty-two degrees above the horizon to fifty 
degrees above the horizon. The same angular divisions are used to divide the space below 
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the horizon. The row size divisions are nonlinear and widest at the horizon since users are 
usually most interested in views near the horizon. However, different angular divisions, 
such as linear angular divisions, may also be used. 

After dividing each unwrapped panoramic image into tiles, a tile fetching 
program (440 and 443) fetches tiles for transmission to the client system. Ideally, the tile 
fetching program only selects the tiles needed to create a view on a client computer system. 
In a preferred embodiment, the client computer system determines which tiles are needed 
and sends a request that specifies which tiles should be selected. The fetched tiles are 
transmitted using a tile streaming program (460 and 463). By transmitting only the tiles 
needed to create a view on the client computer system, the system of the present invention 
limits the amount of information that must be transmitted from a panoramic image source 
server to a panoramic image client display. 

Tile Tiering For Magnification 

To accommodate several different magnification levels, several different sets 
of tile resolution may be created. For example, a low resolution "zoomed back" initial set of 
tiles may be used for panning around a panoramic image. The low resolution initial tiles 
would encompass a large angular view. When an interesting area of the panoramic image is 
selected, a higher resolution set of tiles could then be selected. 

Figures 5D, 5E and 5F illustrate one possible embodiment where a 
panoramic image space has been divided into three tiers of magnification. An initial 
"zoomed back" tile magnification intermediate planar representation is presented in Figure 
5D. The lower resolution intermediate planar representation of Figure 5D is used first. 
When a user wishes to see a portion of a view in greater detail, the client software "zooms 
in" by fetching tiles from the next higher resolution intermediate planar representation 
displayed in Figure 5E. If even greater resolution is needed, the client software "zoonxs in" 
further by fetching tiles from Figure 5F. In one embodiment, the pixel resolution of the 
individual tiles of Figures 5D, 5E and 5F is the same such that the individual tiles take up 
the same amount of screen space but provide successively greater resolution. 

-11- 
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Tilinp Without Transfonnation 

As stated in the beginning of this section, the initial panoraniic image frames 
need not be transformed into an intermediate planar representation before division into 
individual tiles. The intermediate planar representation transformation is usually performed 
since the transfonnation is used to put the image into a format that can be used to quickly 
generate an image for display with minimal processing. However, the raw panoramic image 
frames may immediately be divided into tiles and transmitted. 

For example. Figure 5G illustrates a raw panoramic annular image frame 
from the panoramic camera system of Figure 1. Before transmitting, the raw panoramic 
annular image is divided into tiles. In the embodiment of Figure 5G, the raw panoramic 
annular image is divided into square tiles that are small near the center since a small tile 
covers a wide image area in the annular format. The middle part of the annular disk is 
divided into larger tiles. The outside of the annular disk is divided into smaller tiles since a 
narrow strip of tiles is needed when constructing a view that uses information from the edge 
of the annular disk. Thus, the tiles are spatially variant. 

Tile Positional Selection 

Figure 6A illustrates an example of a client system view into a panoramic 
image. Specifically, client view 630 represents a view into an unwrapped panoramic image 
610 requested by a client computer system. The view is distorted since the view is 
illustrated within the spherical coordinate space. To construct the view 630 at a client 
system, a server (such as server 400) transmits tiles 621, 623, 625, and 627. Each tile 621, 
623, 625, and 627 is. compressed before transmission to further reduce the amount of 
information that needs to be transmitted. Figure 6B illustrates how the four tiles 621, 623, 
625, and 627 from the server are used to construct client view 630 on a client computer 
system. (The four tiles 621, 623, 625, and 627 appear distorted since the tiles are displayed 
in the rectangular projection coordinate space of Figure 6B.) 
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Referring back to Figure 6A, if a user moves the view 630 further to the 
right, then the view will cease to need tiles 621 and 623 and will instead begin to need tiles 
628 and 629. To accommodate for such situations, the client system begins requesting tiles 
that are just slightly out of a current view when a view nears the edge of a tile in case the 
user moves further in that direction. Thus, in the example of Figure 6A, the client system 
would begin requesting tiles 628 and 629 when the view shifts a little further to the right. 
Thus, the tile fetching program in the server would fetch and transmit tiles 628 and 629 to 
the client system. Then, if the client view 630 progresses even further to the right, tiles 623, 
628, 627, and 629 would be used to construct the shifted client view in the client system. 

In a preferred embodiment, the client computer system is responsible for 
determining when the out of view tiles are needed and when they should be transmitted. 
The client system makes this determination by examining factors such as how fast the user 
is moving the view, the latency of the connection with the server, and the parameters that 
define the views. The client computer system uansmits a list of desired tiles to the server 
wherein the list of desired tiles define which tiles should be transmitted by the server to the 
client computer system. 

The movement anticipation procedure described in the previous two 
paragraphs works well for small incremental client view shifts (which most view changes 
comprise). However, if a user makes a sudden quick view change, the needed tiles will not 
be available. For example, referring to Figure 7. a user may quickly shift a view from a 
first view position 730 to a second view position 740. The first view position 730 was being 
rendered with tiles 721, 723, 725, and 727 that were being sent a server. The new second 
view position 740 requires tiles 751, 753, 755, and 757 which are not being transmitted. 

To handle such sudden view changes, the present invention occasionally 
transmits a highly compressed version of the full panoramic image intermediate planar 
representation. The highly compressed panoramic image intermediate planar representation 
is only transmitted once every 30 or so normal frames. When a fast movement occurs, the 
client renders a view from the most recently received highly compressed panoramic image 
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intennediate planar representation. This newly rendered view will not be in the normal 
resolution since it is being rendered from the highly compressed panoramic image 
intermediate planar representation. Simultaneously, the server will begin transmitting the 
tiles needed at the new view location. 

In the example of Figure 7A, the server will begin transmitting tiles 751, 
753, 755, and 757. When tiles 751, 753, 755, and 757 are received by the client computer 
system, the image rendered from the most recently received highly compressed panoramic 
image intermediate planar representation will be replaced with an image rendered from tiles 
751, 753, 755, and 757. Thus, the new view image will become clearer once the server 
"catches up" to the users sudden view change. 

A sudden movement may not move the view completely away from the 
currently available tiles. For example, a movement may place the view partially on tiles that 
are available and partially on tiles that are not available. In such a situation, the portion of 
the view that is located on available times can be rendered at the resolution of those tiles and 
the remainder of the view will be rendered using the highly compressed version. Similarly, 
if the user is "zoomed in" on high resolution tiles and the user moves quickly away from the 
currently available high resolution tiles, the view may be constructed using the tile from the 
next lower resolution tier if that tile is still available. 

Figures 8 and 9 provide detailed flow diagrams that fully describe how the 
electronic image distribution system of Figure 4 operates. Figure 8 illustrates a flow chart 
describing one embodiment wherein a panoramic motion image server system and client 
system negotiate a connection. However, it should be noted that in other embodiments* a 
standard connection can be used such that there is no negotiation of connection parameters. 
Figure 9 describes how a server system transmits image tiles to a client system. 

Client-Server Motion Panoramic Image Connection Negotiation 

Figure 4 illustrates one embodiment of a motion panoramic image client- 
server system wherein the bandwidth is conserved by dividing panoramic images into tiles. 
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II should be noted that there is often not a "standard" server and client environment. 
Specifically, the clients, the server, and the network in a client-server environment may all 
vary depending on the circumstances. The capabilities of each different panoramic server 
system and each different panoramic client system would vary depending upon the systems 
specifications including the computer make, the processor type, the processor generation, 
the amount of random access memory available, the bus speed, and the operating system. 

To most efficiently distribute motion panoramic images from a motion 
panoramic image server to clients, the present invention has a client-server negotiation. The 
server system determines the characteristics of the client before determining how the client- 
server system will operate. Figure 8 illustrates a flow diagram describing the method of 
negotiating the client-server motion panoramic image connection. 

Referring to Figure 8, a client initiates a connection to a panoramic image 
server at step 810. Next at step 820, the panoramic image server requests information thai 
describes the display characteristics the client. Specifically, the server may request the 
client's frame buffer size, the client's frame buffer color characteristics, the size of the 
panoramic image view window, and other information relevant to the display of images on 
the client system. The motion panoramic image server may also request information 
describing the processing abilities of the client. For example, the panoramic image server 
may request the processor type, the processor speed, and the amount of main memory 
available. The client responds to the server's by sending this information to the motion 
panoramic image server at step 830. 

At step 840, the motion panoramic image server uses the collected 
information to determine how the server will serve the client. One specific parameter that 
the server must decide is how to divide the unwrapped panoramic intermediate planar 
representation into tiles. Specifically, the size of the individual tiles will be determined. 
The motion panoramic image server may also determine the video frame-rate, a type of 
compression to use, and a processing-task division based upon the network connection 
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speed and quality, the capabilities of the client system, and the current and anticipated 
processing load of the image server. 

After determining the parameters of the motion panoramic image connection, 
the motion panoramic image server transmits the connection parameters to the client system 
at step 850. The client system will use the parameters to determine how the client system 
will display motion panoramic images. 

After the client and server have negotiated the motion panoramic image 
connection, the client begins requesting motion panoramic image information from the 
server system at step 860. In response to the requests, the server begins transmitting motion 
panoramic image tiles to the client system at step 870. Since the connection to each client is 
uniquely calibrated, the server will transmit panoramic image information to each different 
client in a manner that is specific to that particular client. The server will continue 
transmitting motion panoramic images until the client requests the server to stop sending 
images at step 880. 

Motion Panoramic Image Server Operation 

Figure 9 describes how the motion panoramic image server transmits the 
information needed to construct a view within a motion panoramic image on a client system. 
Thus, Figure 9 describes in detail what occurs during the steps 860 and 870 of Figure 8. 

Referring to step 910 of Figure 9, the first step that the motion panoramic 
image server must perform is transforming a panoramic image frame into one or more 
intermediate planar representations. The exact method of transforming a panoramic image 
frame is dependent on the type of camera system used. For example, a system built using 
the panoramic camera system of Figure 1 would perform a geometric transformation to 
transform the annular panoramic image frame into the selected intermediate planar 
representation. If a fisheye lens panoramic camera system were used, a different geometric 
transformation would transform the distorted fisheye image into the desired intermediate 
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planar representation. All subsequent actions are performed on the transformed intermediate 
planar representation. 

In an embodiment with magnification, the source panoramic image data is 
transformed into more than one intermediate planar representations of different resolutions. 
For example, the source panoramic image data can be transformed into low, medium, and 
high resolution intermediate planar representations as illustrated in Figures 5D, SE and 5F. 

After the panoramic image frame has been transformed into an intermediate 
planar representation, the method proceeds to step 920. At step 920, the method determines 
if a highly compressed version of the full panoramic image frame should be transmitted. As 
previously set forth, a highly compressed version of the full panoramic image frame is 
transmitted occasionally to handle sudden movements of the clients view. In one 
embodiment, the motion panoramic image server transmits a highly compressed version of 
ihe full panoramic image frame during first and every Nth subsequent frame. Thus, during 
the first and every Nth subsequent frame, the method proceeds to step 970 where the full 
panoramic image frame is compressed. Then, at step 980, the motion panoramic image 
server begins transmitting the highly compressed version of the full panoramic image frame. 
The transmission of the highly compressed version of the full panoramic image frame may 
occur all at once. Alternatively, the highly compressed version of the full panoramic image 
frame may be transmitted concurrently along with individual image tiles such that individual 
image tiles are not delayed. 

Next, at step 930, the one or more intcmiediate planar representation frames 
are divided into individual tiles as illustrated in Figures 5D, 5E and 5F. The size of the 
individual tiles is the size that was negotiated during the connection process described in the 
flow diagram of Figure 8. 

After the frame has been divided into individual tiles, the server then fetches 
and transmits the tiles needed to create the client's view at step 940. Each tile is compressed 
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before transmission to conserve bandwidth and decrease latency. The client will use the 
transmitted tiles to construct a image. 

At step 950, the server method determines if the client system is done 
viewing the motion panoramic image. If the client is not done, the method proceeds back to 
step 960. 

Motion Panoramic Image Server Operation 

Figure 10 illustrates a flow diagram describing a method that a client system 
may follow to receive information from render an image. It should be noted that Figure 10 
only illustrates one possible embodiment of a client .system and thai signiTicant variations 
may be created. For example, several of the steps described in Figure 10 such as receiving 
tiles and rendering an image may be performed concurrently. Alternatively, less processing 
could be performed in the client system and more processing could be performed in the 
server. 

Referring to step 1010 of Figure 10, the client receives any and all request 
panoramic image tiles and highly compressed full panoramic images transmitted by the 
server. At step 1020, the client attempts to render an image for the current view position. 
Ideally, the view will be rendered with the tiles needed for the view. However, if the client 
does not have the desired tiles, then the client renders an image of the current view using the 
available tiles for the view, tiles from lower resolution tiers, and the most recently received 
version of the highly compressed full panoramic image. 

After rendering an image frame, the client system accepts input from a user at 
step 1030. The client system may accept input such as directional commands from a 
computer mouse that pan the view within the panoramic image. The client may also accept 
commands that zoom-in and change the vertical viewing angle. 

The client system then processes the user input to the server system at step 
1030 to determine a next view position. At step 1040, the client system determines if the 
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input specifies thai the user is done viewing the panoramic image information. The client 
panoramic image rendering program will terminate when the user no longer wishes to view 
the panoramic image. 

At step 1050 the client system determines the frame tiles that are needed to 
create the new view position. The client system also determines if it should request tiles that 
are just out of the current view but are likely to be needed in the future. Specifically, the 
client determines if the current view is near the edge of a tile. If the client's view is near the 
edge of a tile, then the client will request tiles that are just outside of the client's current 
view. The tiles just out of view are requested such that if the client moves the view beyond 
the edge of the tile, then the client will be able to create a view without having to request 
and wait for the server to then send the needed tile. Thus, the client anticipates small view 
movements and is prepared to generate a view. However, if the user makes a sudden large 
view change, then the client rendering engine can always fall back to the highly compressed 
version of the full panoramic image. 

Furthermore, at step 1050, the client sy.stem determines if any currently 
requested tiles arc no longer needed. Tiles that are no longer needed consume valuable 
bandwidth such that the client .system should unsubscribe from tiles that are no longer 
needed. 

After determining the tiles needed for the current view, the client .system 
determines at step 1060 if the needed tiles have already been requested and if there arc no 
unnecessary tiles on request. If the needed tiles have already been requested and there are 
no unnecessary tiles on request, then the client system does not need to change the tile 
subscriptions. However, if new tiles are needed or currently subscribed tiles are no longer 
needed, then the client system .sends a request to the server at step 1070 that changes the tile 
subscription. 

At step 1080 the client system determines if any new frame tiles or 
compressed panoramic image frames have been received. If new frames have been 
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received, then the method proceeds back to siep 1010 to accept the new frames. Otherwise, 
the method proceeds back to step 1020 to render a new view image using the currently 
available tiles and the highly compressed full panorama image information. A different 
image may be generated from the same frame information if the user has shifted the view. 

Overlapping Tile embodiment 

In one proposed embodiment, the panoramic image intermediate planar 
representation is divided into overlapping tiles. An example of this is provided in Figure 
11, By dividing the intermediate planar representation into overlapping tiles, a degree of 
Hysteresis is provided such that small back and forth view changes will not cause the client 
system to oscillate requests for new tiles. 

A Second Motion Panoramic Image Distribution System Embodiment 

The selective tile transmission system described in the previous section 
provides a useful method of delivering motion panoramic image information in a unicast 
environment where each client receives a unique individual data stream. However, since the 
motion panoramic server must send out a unique data stream to each client, there will be a 
significant amount of redundant information being transmitted. To reduce the amount of 
redundant information being transmitted, the present invention also introduces a .second 
multicast motion panoramic image serving system. 

Figure 12 illustrates a multicast motion panoramic image serving system 
constructed according to the teachings of the present invention. In the system of Figure 12, 
a motion panoramic camera system 1205 delivers a series of high resolution panoramic 
images across a high bandwidth link 1202 to a motion panoramic server 1200. A Panoramic 
Camera Interface 1210 handles the communication with the motion panoramic camera 
sy.stem 1205. A transformation unit 1217 transforms the raw panoramic image frames into 
an intermediate planar representation. As previously set forth, two possible intermediate 
planar representations are cylindrically encoded intermediate planar representation and 
spherically encoded intermediate planar representation. 
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After a panoramic image frame has been transformed into an intermediate 
planar representation, the intermediate planar representation is divided into individual tiles 
by unit 1241. After dividing the intermediate planar representation into individual tiles, a 
set of tile streaming units 1261, 1262, 1263, . . . 126n transmit the tiles on multicast 
channels 1293 on a high bandwidth network 1290. Specifically, each tile position is 
assigned its own multicast channel. 

To display a view within a motion panoramic image, each client system 
subscribes to the multicast channels that carry the tiles needed to construct the desired view. 
For example, client system 1295 needs the two tiles delivered by tile streaming units 1261 
and 1262 such that client system 1295 subscribes to the multicast channels transmitted by 
those tile streaming units. 

Figure 13 provides a flow diagram that describes one possible method that a 
client system may follow to render a view from a motion panoramic server constructed 
according to the teachings of Figure 12. Initially, at step 1310, the client system subscribes 
to the multicast panoramic tile channels necessary to render a view from an initial view 
location. Routers and switches that couple the client system to a main multicast backbone 
will perform the necessary operations in order to begin delivering the subscribed multicast 
panoramic tile channels. 

Next, at step 1320, the client system will render an image of the current view 
using the information from the subscribed multicast panoramic tile channels. It should be 
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noted that other client systems will share the same multicast channels such that bandwidth is 
conserved. 

At step 1330, the client system accepts input from the user. The client system 
may accept input such as directional commands from a computer mouse that pan and tilt the 
user's view within the motion panoramic image. 

After accepting the user's input, the client system analyzes the user's input. 
If the user has indicated that he is done viewing the motion panoramic image at step 1340, 
then the client system unsubscribe to all the multicast tile channels will slop displaying 
images as set forth in step 1370. 

If the user's input changes the position of the view within the motion 
panoramic image, then the client system determines the tiles that will be necessary to render 
the new view position at step 1350. Specifically, referring back to Figure 5, the client 
system determines which files comprise the current view location. At step 1360, the client 
determines if only the same tiles that are already subscribed to arc needed. If .so, then the 
method proceeds back to step 1320 to render a sub.sequent image view frame. 

If different tiles are needed, then the client system proceeds to step 1310. At 
step 1310, the client .system subscribes to the multicast tile channels needed to render the 
current view. 



-22- 



wo 00/08889 ^ ^ PCT/US99/01104 



To limit the number of multicast channels used, the number of tiles that the 
panoramic image is divided into should be kept small. Figure 14 illustrates one possible 
way of dividing the original source panorama. In the embodiment of Figure 14, the source 
panorama is divided into four quadrants: an upper quadrant, a left quadrant, a right quadrant, 
and a lower quadrant. (The quadrant are illustrated using dotted lines.) Each quadrant 
becomes the source of a multicast channel. Thus, only four multicast channels are required. 

To handle pans of the user's view, the information supplied for each quadrant 
should overlap the other quadrants such that only one multicast channels needs to be 
subscribed to. Ideally, the overlap area will be at least as large as a client view. Referring to 
Figure 14, a set of overlapping quadrants are displayed: overlapping upper quadrant 1410, 
overlapping left quadrant 1420, overlapping lower quadrant 1430, overlapping right 
quadrant 1440. 

If very fast pans occur, the multicasting embodiment may also include a 
highly compressed full panoramic image channel. The highly compressed full panoramic 
image channel would occasionally transmit a highly compressed version of the fuU 
panoramic image such that a view in any direction could be created. 

To divide the processing that must be performed, multiple servers may be 
used. For example, a first server could generate and serve the multicast channel carrying the 
overlapping upper quadrant 1410. a second server could generate and serve the multicast 
channel carrying the overiapping left quadrant 1420. and so on. 
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Figure 15 graphically illustrates how network bandwidth is conserved using 



a multicast type of system. Specifically, a panoramic image server transmits all the tile 
channels on a multicast backbone. Various routers coupled to the multicast backbone only 
route multicast channels subscribed to by clients connected to their subnetworks. By routing 
only the multicast channels needed by clients connected to their subnetworks, the routers 
limit the tile information carried by the subnetworks. 



distributing motion panoramic images. It is contemplated that changes and modifications 
may be made by one of ordinary skill in the art, to the materials and arrangements of 
elements of the present invention without departing from the scope of the invention. 



The foregoing has described methods and apparatuses for electronically 
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CLAIMS 

We claim: 

1 . An electronic image distribution apparatus, said apparatus comprising: 

a motion panoramic image source, said motion panoramic image source providing 

successive frames of panoramic images; 
a division unit, said division unit dividing said successive frames of panoramic 

images into tiles, said tiles being spatially variant; and 
a transmitter, said transmitter transmitting a first set selected tiles to a first client 

system to construct a view in said client systenL 

2. The electronic image distribution apparatus as claimed in claim 1 wherein 
said transmitter divides said tiles into a size also dependent upon said first client system. 

3. The electronic image distribution apparatus as claimed in claim 1 further 

comprising: 

a transformation unit, said transformation unit transforming said successive frames 
of panoramic images into an intermediate planar representation before passing 
said frames to said division unit. 
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4. The electronic image distribution apparatus as claimed in claim 3 wherein 
said intermediate planar representation is a spherical projection representation. 

5. The electronic image distribution apparatus as claimed in claim 3 wherein 
said intermediate planar representation is a cylindrical projection representation. 

6. The electronic image distribution apparatus as claimed in claim 3 wherein 
said transformation unit creates more than one intermediate planar representation wherein 
each intermediate planar representation has different resolution levels. 

7. The electronic image distribution apparatus as claimed in claim 1 wherein 
said panoramic image frames comprise an annular panoramic image format. 

8. The electronic image distribution apparatus as claimed in claim 1 wherein 
said transmitter compresses said tiles. 
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9. The electronic image distribution apparatus as claimed in claim 1 further 

comprising: 



10. The electronic image distribution apparatus as claimed in claim 9 wherein 
said compression unit compresses every Nth of said successive frames of panoramic images 
into said highly compressed format. 

1 1. The electronic image distribution apparatus as claimed in claim 1 wherein 
said client system requests said first set of selected tiles. 

12. The electronic image distribution apparatus as claimed in claim 1 wherein 
said transmitter further transmits a second set of selected tiles to a second client system. 



a compression unit, said compression unit occasionally compressing one of said 
successive frames of panoramic images into a highly compressed formal and 
transmitting said highly compressed format to said first client system. 



13, The electronic image distribution apparatus as claimed in claim 12 
wherein said second client system requests .said .second set of selected tiles. 
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14. The electronic image distribution apparatus as claimed in claim 1 wherein 
said tiles overlap each other. 



15. The electronic image distribution apparatus as claimed in claim 1 wherein 
transmitter transmits more than just the tiles needed to render said view in said client 
system. 



16. A method of distributing electronic images, said method comprising the 

steps of: 

capturing a panoramic image frame; 

dividing said panoramic image into a plurality of tiles, said tiles being spatially 
variant; and 

selectively transmitting a subset of said plurality of tiles to a first client system. 

17. The method of distributing electronic images as claimed in claim 16 
further comprising: 

transforming said panoramic image frame into an intermediate planar format 
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18. The method of distributing electronic images as claimed in claim 17 
wherein said intermediate planar representation comprises a spherical projection 
representation. 

19. The method of distributing electronic images as claimed in claim 17 
wherein said intermediate planar representation comprises a cylindrical projection 
representation. 

20. The method of distributing electronic images as claimed in claim 17 
wherein transforming said raw panoramic image frame comprises transforming said raw 
panoramic image frame into more than one intermediate planar format wherein each 
intermediate planar format has a different resolution level. 

21. The method of distributing electronic images as claimed in claim 17 
wherein .said tiles are divided into a size further dependent upon said first client system. 



22. The method of distributing electronic images as claimed in claim 16 
herein said first client system specifies which tiles must be transmitted to render a vie 



said client system. 
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23. The meihod of disiribuiing electronic images as claimed in claim 16 
wherein said clienl requests outside tiles just outside of said view in case said client system 
needs said outside tiles for a future view. 

24. The method of distributing electronic images as claimed in claim 16 
further comprising 

transforming said raw panoramic image frame into a highly compressed intermediate 

planar format frame; and 
transmitting said highly compressed intermediate format frame to said first client 

system. 

25. The method of distributing electronic images as claimed in claim 16 
wherein said subset of said plurality of tiles are compressed before being transmitted. 



26. The method of distributing electronic images as claimed in claim 16 
wherein said plurality of tiles overlap each other. 
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27. The method of distributing electronic images as claimed in claim 16 
further comprising: 

selectively transmitting a second subset of said plurality of tiles to a second client 
system. 

28. An electronic image distribution apparatus, said apparatus comprising: 
a motion panoramic image source, said motion panoramic image source providing 

successive frames of panoramic images; 
a divider, said divider dividing said successive frames of panoramic images into a set 

of individual tiles; and 
a transmitter, said transmitter transmitting tiles on multicast channels. 

29. The electronic image distribution apparatus as claimed in claim 28 further 

comprising. 

a transformation unit, said transformation unit transforming said successive frames 
of panoramic images into intermediate planar format; 



wherein said intermediate planar formal comprises a spherical projection representation. 



30. The electronic image distribution apparatus as claimed in claim 29 
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31. The electronic image distribution apparatus as claimed in claim 29 
wherein said intennediate planar representation comprises a cylindrical projection 
representation. 

32. The electronic image distribution apparatus as claimed in claim 28 
wherein said successive frames of panoramic images comprise annular panoramic image 
frames. 

33. The electronic image distribution apparatus as claimed in claim 28 further 

comprising: 

a first client system, said first client system subscribing to a first set of multicast 
channels needed to render an image. 

34. The electronic image distribution apparatus as claimed in claim 33 further 

comprising: 

a second clienl system, said second client system subscribing to a second set of 
multicast channels needed to render an image. 
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35. The electronic image distribution apparatus as claimed in claim 28 
wherein said transmitter compresses said individual tiles. 

36. The electronic image distribution apparatus as claimed in claim 28 
wherein said multicast channels comprise multicast su-eams on the Internet. 

37. An electronic image distribution method, said method comprising: 
capturing successive frames of panoramic images; 

dividing said intermediate format frames into a set of individual tiles, said tiles being 

spatially variant; and 
transmitting said individual tiles on individual multicast channels. 

38. The method as claimed in claim 37 further comprising: 
transforming said successive frames of panoramic images into an intermediate planar 

representation. 

39. The method as claimed in claim 37 further comprising: 
subscribing a first client system to multicast channels needed to render a first view 

into said panoramic image on said first client system. 
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40. The method as claimed in claim 39 further comprising: 
subscribing a second client system to multicast channels needed to render a second 

view into said panoramic image on said second client system. 

41. The method as claimed in claim 37 wherein said individual tiles are 
compressed before transmitting. 



42. The method as claimed in claim 37 wherein .said multicast channels 



comprise multicast streams on the Internet. 
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ABSTRACT : PROBLEM TO BE SOLVED: To tFeraforootor Image Infbmialkxi or the like aiKj to allows 
receiver side to reproduce the Infonna^m without Impoalr^ 
transinitter and uaeieee communlcalhx^ in a netwoilc 



SOLUTION: An Image server 2 sends ooior infOmudion or the like and locadon Inforniadon 
to specify the tocatton of proflle infom^tton. corresponding to the ooiorlmage infbmfiatkx) 
or object Infomnatton to a network. In m Image display device 3. a reoeptton means 30 
rsceivee the cotor Image lnfomiatk)n or the Oke and the kxaUon Infomtation, a call means 
34 calls the proflle Intomnation. oonespondbig to the kscatkxi mformatkxi from a proflle 
storage meana 10 via the netwofk. A color conBCtkxi means 35 usee the profile 
infomudton to oonect a cokx of the oo^of image infomudkxi c the Ilka and a reproduction 
means 36 reproduoea the InfMrnatton oonectad. 
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